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(57) Abstract: An ultrasound imaging system superimposes sectional views 
created from volumetric ultrasound data and the location data for an inter- 
vention device, such as a catheter. The position of an interventional medical 
device may be shown, in one or more views, relative to organs and tissues 
within a body as the interventional device is moved. The interventional de- 
vice positional data is updated continuously and is superimposed on tissue 
images that may be updated less frequently, resulting in real-time or near 
real-time images of the interventional device relative to the tissues. The su- 
perimposed images permits medical personnel to perform procedures such 
as angiograms with minimal or no exposure of patients to x-rays and con- 
trasting dye. The look and feel of the familiar fluoroscopy -like imaging may 
be maintained, or a three dimensional real-time, or near- time projected im- 
age of the intervention medical device relative to an organ or tissue may be 
. provided. 
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ULTRASONIC IMAGER 
BACKGROUND OF THE INVENTION 

Field of the Invention 

The present invention relates to ultrasonic imaging. More particularly, the 
5 present invention relates to creating sectional views from volumetric ultrasound data 
* and superimposing positional data for an interventional device, such as a catheter. 

Background of the Invention 

Many medical procedures involve inserting a catheter of other interventional 
- device, into a patient for a variety of purposes. One such procedure is cardiac 
10 . ..catheterization for an angiogram. 

To observe the condition of blood vessels within the heart, radio-opaque dye 
is injected into the bloodstream through the catheter and an x-ray image is taken. 
Typically, this procedure involves inserting a catheter into an artery in -the patient's 
eroin area and guiding the catheter through the arterial system and the aorta to the 
15 heart. In order to position the catheter and monitor its location within the patient,' one 
or more x-ray images may be taken prior to "injecting dye for the angiogram. 

A catheter may be directly visible in an x-ray image, however, a heart typically 
is not. In order to determine the location of the catheter relative to the heart, dye may 
be injected through the catheter into the bloodstream. Tne dye acts as a contrasting. 
20 agent allowing the heart to be located, by imaging' the dye flowing through it. The 
image of the heart is then "frozen" at a point in time when the dye location (and 
concentration), provides the best image. Often, this snapshot image of the heart is ; 
traced on a transparent overlay which'is then superimposed on a display monitor. After 
the dye concentration decreases, the overlay may be used. to track.the catheter location 
25 and movements relative to heart, as indicated' by the overlay.'" There are two 
disadvantages to this process. First, the image of the heart, made visible with the dye, 
is.only temporary, requiring the overtey to track the catheter movement.. Second, if the 
patient's position changes, the heart imaging must be repeated.. 
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In addition to locating a catheter relative 10 a bean for an angiogram, mere are 
many other medical procedures where an interventional device is inserted into a 
patient. lamest, if not all, such procedures, locating and positioning the interventional 
device within a patient is extremely important. 
5 Multiple views, or x-ray orientations, of. a patient are often desirable to 

visualize objects and locations in three dimensions. Although there are many possible 
combinations of such views, some are favored by medical personnel for particular 
procedures. For cardiac imaging, the right anterior oblique (RAO) and the left anterior 
oblique (LAO) views are common. Figure 1 shows a schematic sectional depiction 
10 of a prone patient 1 0 and x-ray sources 12 and 14. The RAO and LAO views are taken 
from the patient's right and left/respectively, of the anterior (front) of the.heart. The 
oblique angle, shown as angles 16 and 18, is.the measure of how far off the vertical an 
' x-ray source is oriented.. The multiple views are often taken with a single x-ray source 
that is repositioned between positions 12 and 14. An imaging system that readily 
15 presents RAO and LAO view allows medical personnel to view images in a femiliar 
- format, as this is what is presently standard in the art. Multiple preferred views for 
other procedures might similarly be pre-defined. 

There are problems with present practices; First, x-rays and fluoroscopy 
produce radiation, and the effect of this radiation over the long term may be harmful. 
20 This is especially true for the medical personnel subjected to repeated radiation 
exposures over the course of an entire career. 

In addition, the radio-opaque dye may harm the patient. For example, an 
angiogram is often performed on patients with serious heart problems. Injecting 
chemicals that may cause sensitivity, or an allergic reaction, directly into the heart of 
25 such a patient may cause a serious problem, and there is the possibility of kidney 
damage from the dye. 

Finally, x-ray techniques require complex: x-ray equipment and the costly 
' overhead associated with such equipment. While this may not.be a primary concern 
in a well equipped , modem hospital, is of concern in less, developed or remote 
20 locations. 

Ultrasound based imaging methods offer some potential advantages overx-ray 
• based imaging methods. Ultrasound methods do. not expose ihe paticiirormedical 
personnel to radiation and, may. reduce or eliminate the need for costly x-ray 
equipment. Also, the ability of ultrasound methods to directly image soft tissue and 
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blood vessels,. wiihout'the need for radio-opaque dye as a contrasting agent, eliminates ■ 
the potential dye-related harms to" a patient 

It would be desirable to have an imaging system that could eliminate,- or serve 
as a partial substitution for, x-ray and fluoroscopy methods of imaging in procedures 
• 5 such as angiography and tissue biopsies. Preferably, such a system would present 
medical personnel with multiple views that are displayed simultaneously. 

• * U.S. Patent No. 4, 1 73,228, issuedto.Steenwyk and Childress on November 6, ■ 
1 979, describes a catheter locating device ("the '228 patent"). The 4 22S patent uses an 
induction coil adjacent to. the catheter tip, and a remote sensing device to monitor the 
10 amplitude and phase of signals induced in the coil,as a means of detecting the catheter 
location: However, an electrical signal from an induction coil is not well suited for 
detection by ultrasound imaging equipment. The f 228 patent does teach one technique 
for locating a catheter, but it lacks' the ability to directly map that location to a 3-D 
ultrasonic image of tissues within the body. Instead, such a device determines a 
15 "catheter location relative to a position on the. skin surface and a depth estimate, based 
bathe magnitude of the signal received at. the skin surface. It would be desirable to . . 
locate the catheter relative to internal body tissues, not the skin. 

US. Patent No. 5,515,853, issued to Smith, et-al. on May .14, 1996, describes 
a 3-D ultrasound tracking system based on triangulation ("the '853 patent tT )- Using a 
20 nenvorkofatleastfourpiezoelectrictransducere^ 

personal computer (PC) as a digital controlier;'the '853 patent accurately . measures the 
relative distances between" the transducers. However, like the '228 patent discussed 
above, an apparatus. based" on the '853 patent lacks. the ability to map the location of 
a catheter to a 3-D ultrasonic image of tissue. within the body. Instead, the reference 
25 frame for catheter location information- is the location of . other piezoelectric 
transducers. For example, a series of transducers contained in-a chest.harness around 
a patient riiay be used to triangulate the position and orientation of a catheter relative 
to the chest harness, by measuring the distances to transducers mounted in the catheter. 
The '853 patent does briefly mention the possibility of "overlaying" transducer location 
30 ■ information on a video loop, created from x-rays or ultrasound, to facilitate 
visualization of the catheter location. However, there .is no teaching of how this 
overlay process" might' be performed.*' -in contrast, ah embodiment. of the present 
invention is directed to describing the catheter location relative to tissue within the 
body in one or more 2-D views. 
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U.S. Patent No.'s 5,817,022 and 5,868,673, ("the '022 and 4 673 patents") . 
issued to Vesely on October 6, 1998, and on February 9, 1999, respectively, are both 
continuations-in-part from the application that became the *853 patent. Both also 
claim 3-D ultrasound tracking of interventional medical instruments by .triangulating 
5 between a transducer attached to the medical instrument and a network of transducers 
either' inside a patient's body or on the body surface. It would be desirable to simplify 
the apparatus- so that a network of transducers is not required. 

• The 4 022 patent locates a medical instrument, relative to the network of 
transducers in 3-D, and then displays a 2-D ultrasound image within the 3-D coordinate 
10 system. The 1 022 patent describes a "imaging modality system", that may acquire 2-D, 
3-D or 4-D. image data, an "image registration system" to register the position the 
[medical] instrument within the image data and a "user interface" to perform a desired 
function/such as selecting a particular view for display. However, there are no detailed, 
descriptions' of these elements, or teachings which would permit one of ordinary skill 
15 . in the art to create a system combining such elements. • 

The '673 patent describes a system for performing surgery while tracking the 
location of a medical devicerelative to a network of transducers, on or within the body. 
The medical device location is then mapped to the network of transceivers, but not 
directly to .any portion of the body tissue. 
20 Ultrasound image "data for an object, such as a heart, . consists of the 

"echodensity" sampled at multiple locations within an object.- This echodensity can 
be thought of as somewhat equivalent to the color or intensity of a visual image. The 
. echodensity is a . function of the sound velocity within, the tissue, as well as the tissue 
density. Although a3-D image of echodensity, with continuous gradations along three 
25 •. dimensions, may lack the clarity of the sharp edges in a solid model, the gradations 
• contain a significant amount of information about the tissue. A solid modeling .. 
."' approach to presenting a 3-D ultrasound image typically uses a threshold value. That 
. is, all echodensity values are divided into two bins, those above and those below the 
threshold. Values above the. threshold are opaque and those below are transparent; 
30 Tnis threshold technique adds clarity to the wide variations of the "murky" echodensity 
. 'data, by distinguishing solid features within the data, at the expense of the detail stored 
in those many values. 

TomTec Imaging Systems, formerly of Boulder, CO, now TomTec Irnaging 
• Systems GmbH of Munich, Germany, produced a video tape describing a 3-D 

-4- 



BNSDOCID: <WCU_Ol5e359AlJ .>. 



SUBSTITUTE SHEET (RULE 26) 



WO 01/58359 PCT/US01/00957 

•ultrasound" imaging system ("the TomTec. system"). TornTec Imaging systems, 
Clinical Three-Dimensional Echocardiography, VHS/NTSC,.19:20 5 040-012.0, 07-96 
(video tape). The TomTec system: creates a 3-D cardiac image, with, the use of a 
ultrasound transducer in the form of an endoscope, and produces an image of the 
5 surfaces of the heart, essentially a solid model. That- is, the surface of the heart is 
•displayed without any information about the internal muscle tissue. The. TomTec 
system does not appear to have been commercially successful. 

- SUMMARY' OF THE INVENTION . 

10 ■ ' . 

The ultrasound imaging system of the present invention superimposes sectional 

views created from volumetric ultrasound data and the location data for an intervention 
' -device, such as, but not l imited 10, a catheter. The position of catheter may be shown, 
. in one or more views, relative io organs and tissues within a body as the catheter is 
15 moved. The catheter positional data is updated continuously and. is superimposed on 
tissue images, that may be updaied less frequently, resulting in real-time or near real- • 
time images of the catheter relative to the tissues. 

. The.superimposed images allow medical personnel to perform proc edures.such 
as angiograms with minimal exposure of patients to x-rays and contrasting dye. 
20 However, the look and feel of the . familiar fluofoscopy-like imaging may be 
. maintained. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a schematic sectional depiction . of a patient and x-ray-equipment. 
Figure 2 is a diagram of a 3-D image space in accordance with an embodiment 

25 of the present invention.- 

Figure 3 is a schematic diagram of point location among regularly spaced 
image data samples in accordance with an embodiment of the present invention. 

Figure 4 is a schematic diagram of irregularly spaced image data samples in 
accordance with an embodiment of the present invention. 
30 Figure 5 is a schematic diagram of a control panel in accordance with an 

embodiment of the present invention. ' 

. Figure 6 is a flow chart of the imaging process in accordance with an 
embodiment of the present invention. 
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DETAILED DESCRIPTION 

The preseni invention is directed at superimposing an image of an 
interventional medical device, such as a .catheter, and a three-dimensional (3-D) 
ultrasound image of an organ or tissues, such as a "heart, and creating one or more 
5 projections from the superimposed image data. The two sets, of image data may be 
updated at the same or different frequencies. The image of the medical- device may be 
updated more frequently to show its current .location, as the device is guided or 
repositioned within a patient by medical personnel. The projections made from the 
superimposed images preferably reflect the current location of the medical device, and 
10 depending on the frequency of updating the image of the organ or tissues, may also 
; show a real-time image of such organ or tissues. 

The- present invention may be. used with a wide variety of interventional 
medical devices, such as, but not limited to, catheters, probes, cannulas, stylets, and 
needles, and is not intended to be limited to use with any particular interventional 
15 medical de vice. For convenience, this disclosure will use the. term catheter, instead the 
more generic term intervention medical device, since the example description is related 
to use of the apparatus and method of the invention in cardiac procedure; however, this 
is not meant to imply any limitation on the scope of the invention. 

The present invention may be usedin connection with ultrasound images of any 
20 organ, portion of an organ, or. group, of organs, and the present invention is not 
intended to.be limited to any particular image subject matter. Tnere are situations in 
which it .may be desirable to use the present invention in conjunction with x-ray 
.methods. For example, if an. ultrasound transducer is positioned so that only locations 
in the vicinity. of the heart, can be imaged, x-ray methods might be used to track a 
25 catheter from its insertion point near the groin until it comes into view with the 
transducer. " 

■ The present invention may also be used with "4-D ultrasound images' 5 . That 
repetitive- sequences of 3-D images in time, such as, but not limited to, images of 
•a heart beating as it goes through one or more cycles. The 4-D ultrasound images may 
30 be. depicted and displayed as repetitive sequences of images, or cine loops. 
Alternatively, a single point in the time sequence may be selected, effectively 
■ transforming the 4-D image into a 3-D image. Preferably, the present invention allows 
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updates of the catheter-location at a frequency independent of the repetitive sequence 
frequency and the frequency of individual image frames within the sequence. Tnis 
disclosure- will use the term "dynamic display" when 4-D image data is distinguished 
from 3-D image data. Otherwise, the term "image data" may apply to either 3-D or 4- 
5 D image data. 

The output of the present invention is one or more projections created from the 
superimposed image data.. Although the projections are typically displayed on one or 
more cathode ray tube (CRT) monitors, the present invention may output the. 
projections to any type .of graphic display device(s), such as a thin film transitor flat 
10 screen display, or to a graphic output file for storage'or further image processing. ' The 
present invention is not intended to be limited to use with any particular type of graphic 
display device or output file format. Multiple projections, typically, taken from 
differing orientations with respect to the image data, are often beneficial in visualizing 
" locations in a 3-D space. An embodiment of the present-invention uses two. standard 
15 views, right anterior oblique (RAO) and left anterior oblique (LAO) as "default" 
projection orientations. 

As the trend' towards faster computer processors continues, eventually the 
present invention will, allow for real-time or near real-time updates of projections 
created. from 3-D image data. At present, the acquisition of image data, and the 
20 projection processing- discussed- below, takes too- long for such real-time updates. 
Given the current constraint on computing power, the.present. invention is preferably 
used in an environment where the patient is immobilized. Tnis permits us e of a single 
set of image data may be used continuously while a catheter is moved" relative to the 
patent. and the image data. 
25 For purposes of the examples described below, references to a dynamic display 

embodiment of the present invention, presumes that the position .of the organs and 
tissue return to the same position and orientation within, a patient, at the end of each 
repetitive sequence, or cycle. That is, the cycle may consist of "local" movement but 
no "global" movement relative to the immobilized patient is anticipated.; In contrast, 
30 movement of the catheter is anticipated, so the. location of the catheter is . updated 
frequently, preferably in real-time. The'updated catheter location is then superimposed 
on the relatively stadc^orgaruoLti^uejmage..da:a..befor.e the projections are created. 
Collecting 3-D ultrasound image'data, while certainly not a trivial procedure, 
. is known to those of skill in the art,. and is not intended to be an element of the present . 
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invention. Rather, the. present invention uses such data, however collected, as input. 
U S: Patent No. 5, 1 8 1,5 1 4, issued to Solomon, et al. on January 26, 1 993, describes a 
transesophageal echo cardiography transducer devi ce which may be used to collect 3-D 
ultrasonic image data ("the '514 patent"). This device, with a rotatable ultrasound 
transducer array, may be inserted into a patient's esophagus and positioned adjacent 
' .to the heart. A motor and drive mechanism, which remains externa] to the patient,- is 
used control. the probe with the transducer array and obtain 3-D image data.- A 
transesophageal' transducer, such as described in the '514 patent, might be used to 
collect cardiac image data. As the present invention is not limited to any particular, 
method or apparatus for collecting image data, iris intended that as advances are made 
in ultrasound data collection, the present invention may utilize such improved image 
data as input. 

Image data will preferably take the form of the 3-D array: 

Or, in the case of image data for a dynamic display,, in the. form of the 4-D array: 

l{x,y,z,t) 

Where / represents the. echodensity, or "intensity /'.of the ultrasound image at a point 
- defined by the 3-D Cartesian coordinates x, y and z. The particular choice of 

orientation for the x, y and z axes does not effect the present invention. Preferably, 

the axis orientation is used consistently. For a dynamic display, i refers to the instant 
in. time representing when the echodensity data was sampled within the repetitive 
sequence. Image data may be collected in different formats, or coordinate systems, and 
then converted to the preferred format using transformation formulae known to those 
of ordinary skill in the art/ 

' In one embodiment of the present invention the / values at each sampled point 
in space will . be : binaiy representations of the echodensity values. The values may be 
in either integer or floating point format. Although integer values are typically better 
• suited to efficient mathematical- algorithms, this efficiency may be effectively lost if 
numerical interpolationmethods using floating point, values are selected for integration. 
The need for numericanmerpolation.and-ihtegrau 

The / values, for monochromatic embodiments of the present invention, willpreferably 
map to a specific.gray scale value that will be used to display the image. For example, 
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eizht bit integer / values may represent 2i5 gray scaie values, from 0 to 255. Floating 
point / values may be grouped in ranges, each of which corresponds to a gray scale 
resolution of the display. The range of gray scaieresoiution values chosen will depend 
on both the characteristics of the display (where higher resolution produces better 
5 images) and the processing speed of the image processing device (where higher 
resolutions require more calculations per pixel). Gray scales for displaying images are 
known to those of ordinary skill in the an. The present invention is. not intended to be 
limited to any particular gray scaie resolution. 

In addition to the number of gray scale values, or gray scale' resolution, the 
10 spatial resolution of the input image data also efizcxs both the quality of the display and 
the number of calculations required. Tnat is, image data sampled at many points-in 3- 
D scace will generally produce a better image than the. same object sampled'at fewer, 
points. Tne trade off being that the additional image sample points may require more 
calculations, which typically requires a more complex and rapid handling system, or 
15 slows the image, processings 

Although techniques for collecting the ultrasound image data are: not within the 
scope of the present invention, and image data is treated as input to the present 
invention, an embodiment of the present, invention- does allow a user to call for an 
update of image data as needed. For. example, if the medical personnel detect that a 
20 patient has shifted, changing the relative position of the obj ect being imaged and the 
catheter, new/ (x, y, z) values may be collected; In another embodiment of the 
present invention, a patient movement that is more than a pre-defined. threshold-amount 
may trigger a call for updating the image data. As long as there is little or no relative 
movement between the object being imaged and the catheter, there is no need to update 
25 the image data. Preferably, the catheter is located using the same apparatus .used to • 
collect the 3-D ultrasound image data, / O, y, O - The format of the catheter location 
data will depend somewhat on the type of catheter used, as well as the associated- 
means of locating the catheter. 

. The catheter. location may correspond to the' direct echodensity data depicting 
30 the image of the catheter, including' any image' enhancements from active. catheter 
devices such as a vibration-mechanism: U.S: PaterrrNav 5\343-,865. issued to Gardineer 
and Vilkomerson on September 6, 1 994. describes a catheter vibrating mechanism 
("the '865 patent"). A vibrating mechanism, such as described in the ' S65 patent, may 
enhance the visibility of a catheter, or other -interventional medical device, that is 
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attached to it in an ultrasound imaging system. The present invention is preferably 
used with such ari active catheter. Potential benefits include faster imaging of the 
location, a. clearer image, and such an active catheter.reduces the required sensitivity 
for the imaging system. Tne present invention is not, however, intended to be limited 
5 to use with .an active catheter, or any particular type of active catheter system. - The 
. image of the catheter location may also be enhanced, for better visibility* through 
techniques such as brightening, increasing the contrast, or enlarging the catheter image 
before superimposing the catheter-location data and the 3-D image data. With some 
embodiments of the present invention, afalse image, or icon, may be used in place' of . 
10 the catheter image data. Such techniques will be further described below. I: is very 
important, however, that the catheter location be matched closely to the coordinate 
syistem of the image data. 

. In-order to create projections of the 3-D image data, one or more .projection 
planes must be defined. Figure 2 depicts a 3-Dspace or volume 22. Volume 22 is the 
15 ponion of space-containing the imaged objects, such as organs within a patient's-bpdy 
cavity, and could take virtually any 3-D shape.. Volume 22 uses (global), axes 24 to 
define a Cartesian coordinate system for orientation, and measurement. A 2-D plane 
26. is selected relative to volume 22, with (local) axes denoted as («, v) . Typically, 

plane. 26 is defined'by a unit vector ri 9 normal to (u, v) . Unit vector, h can be 

20 defined as: 

h = (ri; x -r n 3 y -r Kjz) 
and, because h is a unit vector: 

■ Where x,y and fare unit vectors along the.x 5 j^ and zaxes, respectively, and 
25 '(it i, it:, ri/) are the direction-cosines of h ± Those of ordinary skill in the art will 
recognize that there are many mathematically equivalent ways of defining plane 26 and 
its normal vector h ; 
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Imposing the condition: 

the directions - of -unit, sectors u and v can be denned, using the constant R, as; 

u =R(—ri2, Hi, 0) 

5 where; 

and; 

v = R(—n> Tij y — m «>, n] + n 7 3 ) 

The construction of the projection on plane 26, can be calculated by performing 
10 the integration: 

>(u.v)« jl(x,y,2)dS 

• That is, the image data, / (x, y 7 z), is summed along the. line 

x onto P (z/, v), where the vector ic is an integrating line parallel to it and is defined 

by the equation: 
15 5=x, -r J/i 



The starting point, x 0 , is given by: 



x 0 —uit 4- VV 



= R(— un:— vrzj n*, uni — vnin^ v(nj n 2 2 )f) 

20 and, 

where £ is an integer and C7 is the integration interval used for the numerical 
implementation. 
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S can be thought of as the thickness, or depth, of the image data used to create P ( u, v). 
• In some embodiments of the present invention S, 2nd the starting point x., are set 
' such that the projection is created from only a relatively thin slice of the image data 
I{x, y 9 z). " Preferably,- the depth of the image data used in projection plane 6 is 

5 defined by the user along an axis parallel to h. 

The numerical implementation of the integration used to calculate 
P(u, v) may require interpolation. That is, the vector x and integration interval cr 

- may result in locations that do not directly coincide with any data sample locations in I (x 9 y, z). 

Referring now to Figure 3, which depicts a set of image data values 28, corresponding 
10 to image sample locations within I(x s y 9 z). Values 28 are shown, for clarity, as 

regularly spaced in- three dimensions, although the same principals apply for irregularly 
spaced image values. Location 30- is a point along x used in the calculation of 
P(u, v), that does hot correspond to the location of any value 28. Those of ordinary 

skill in. the art will recognize that there are* many techniques available for assigning a 
15 data value to location 30. A few of these techniques' are, in increasing order of 
complexity, matching the value of the "nearest neighbor", averaging nearby values and 
computing weighted averages. The present invention, however, is. not intended to be 
limited to any particular interpolation technique. 

U.S. Patent No. 5,226,113, issued to 'Glirie, et.aL.6n July 6, 1993, describes 
20 using reverse ray casting to project volumetric images unto an image plane rotated at 
• any arbitrary angle ("the ' 113 patent"). ■ The present invention takes 3-D volumetric 
image data, as one input, and projects 3-D' image data onto one or more image planes, 
which may be rotated relative to the axes of the 3-D image data. The present invention 
may use the projection techniques of the * 1 13 patent, or any another technique, and is • 
25 not intended to be limited to any particular method of projecting 3-D image data to 
create a 2-D view in an image plane. U.S. Patent No..5, 8.65,750, .issued to Hatfield, et 
al, on February 2, 1 999, also described using reverse. ray casting to project volumetric 
' images ("the '750 patent"). Like the '113 patent, the technique described by the '750 
•patent'may be used by ah embodiment of the present invention to project 3-D image 
30 data.. 

The above projection creation procedure may be repeated for each desired 
projection, display. An embodiment of the present invention allows the user to select 

-12- 



BNSOOCID: <WO„_0158359Al_L> 



SUBSTITUTE SHEET (RULE 26) 



WO 01/58359 



PCT/US01/00957 



a first projection, corresponding to either an RAO or a LAO orientation., at a user 
selected number of -degrees off the vertical, with a second projection - (default) 
orient2tion corresponding to the other. Preferably, denning the direction of the second 
projection orientation would.require very Iittie input from the user, and instead would 
5 use translations from the first projection, orientation to orient the second. 

Presenting projections to medical personnel in a familiar format, such as RAO 
and LAO orientations, is merely one capability of the present invention. Unlike x-ray 
• images taken from only one or two orientations, the 3-D im2ge data / (x 7 y y z) used 
with the present invention may create virtually any view(s) a user might select. For 
10 example, a projection could be created of the tissue in front of a catheter, as viewed 
along the axis of the catheter. 

The simplest form for the image data, is equally spaced image-samples in each 
of±cx,'y and 'z directions. Unequally spaced,- or irregularly spaced, samples may 
' also be used with- the present invention although the calculations may become more 
15 complex, and time consuming, irregularly spaced samples may also be pre-processed 
and converted to equally spaced image samples, by any one of a variety of techniques, 
for use by an embodiment of the present invention.' An embodiment of the present 
invention may pre-process irregularly spaced I(x, y, z) data values. • Referring now 
to Figure 4, which depicts an irregularly spaced set of image, data values 28, relative 
20 to a regular spacing location template 32. Data values 2 S may be used to interpolate, 
or create, data samples on template 32. Similar to the interpolation discussed above, 
the present invention is not intended to be limited to any particular method 
interpolating or translating data values 28. Some. projection algorithms may benefit 
from such a pre-processing of irregular image data into regularly spaced locations. 
25 A control panel for an embodiment of the present invention is shown in Figure 

5. Joysticks 34 are used to choose the orientation of one or more projections, and pairs 
of slide bars 36 choose the upper in lower limits of the depth of the projections, S. 
Brightness 38 and contrast 40 controls allow the user to adjust the- display oh one of 
more CRTs or other display devices. Tne present invention is not intended to be 
30 limited to any particular method of allowing a user to choose a projection-orientation 
or control the displays. Although direction cosines are used irrxhis disclosure to 
describe projection pianes.. it is not intended for a user to be restricted to inputting 
orientation information- in that format. 
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In one embodiment of me present invention, tne location oi me catneter is 
superimposed on the image data after the projection(s) have been created. Generally, . 
the creation of projections, P(//, v), from the imasedaia file I(x, y> z) is a relatively 

' "slow process^ Although it is understood that the trend towards faster and less 
5 expensive computer processing will ease this constraint At the present time the 
projections are created less frequently than in real-time. This means that as the catheter 
moves, first superimposing the catheter location and I(x, y, z). and then creating one 
or more projections, P(w, v), may introduce a time-lag in displaying actual. catheter 
location. Unlike the image data file J (at, j>,.z), which typically contains non-zero 

10 data samples at many thousands of locations corresponding to the-.entire volume of the 
imaged organs or tissues, the catheter location data- will typically be relatively "sparse". 
That is, it will contain much fewer non-zero entries than / (x, y, z). An embodiment 
of the present invention exploits the fact that the, catheter . location data is relatively 
sparse compared to I(x, y, z), and superimposes the catheter location directly on the 

15 projection(s). In this way, the changing position of the catheter can be mapped onto 
the projection, without the need to re-create the projection from each data point within 
I(x y y, z) . In most cases, the relatively sparse catheter location data can be directly 

superimposed on projections in much-less time than is required to create the projection 
from I(x, y, z). 

20 ■ Figure 6 shows a flow chart of an embodiment of the present invention where 

" the intervention medical device location is superimposed 42 after the projection(s) are 

created 44, and then the projection with the device location is displayed 46. The 

process- of locating the device position 48 is repeated frequently, preferably at a 

frequency close to the display monitor video refresh rate. 
25 ... .Re-starting the process, by obtaining updated image data 50, may be initiated 

by the user in situations such as where. there is patient movement. With- sufficient 

computer processing power, obtaining updated . image data 50 may be repeated at 

predetermined frequency without user action. .-. 

Tne location of the catheter, is preferably detected by the same ultrasonic 
30 . imaging, equipment that is used to create the. image data, I(x, y, z), using the same 

global Cartesian coordinate system. By.using an active catheter system, such as the 

vibrating mechanism of the. ' 

-14- 



SUBSTITUTE SHEET (RULE 26) 



WO 01/58359 PCT/US01/00957 

'865 patent, the '-catheter image' may De mors easuy aeiectea, ana aisunguisnea trom, 
the relatively passive surrounding environment. An embodiment of the present 
invention may also use one or more filters to further distinguish the active catheter 
image from background. Such filtering techniques are generally known to those of 
5 ordinary skill in the art. 

The catheter image may take one of two preferred formats.. First, it may match 
the format of the input file I(x, y, z)wiih only a relatively limited number of non- 
zero entries. For the catheter image this file will be called J.(x, y, z). In this 
format, L(x, y, z) will contain the echodensity values of the catheter, corresponding 
10 to the actual 3-D image of the catheter, as seen by the ultrasound transducer. As 
discussed above, the J. (.r, y, z) values may be the post-filter echodensity data. 

Alternatively, the catheter image may correspond to one or more sets of 
Cartesian coordinates, representing the coordinates of one or more locations on the 
catheter in 3-D space. That is, ihe actual ultrasound image of the catheter is not used, 
15 only thecoordinates of one or more points that may be used to determine the catheter 
location, and perhaps its orientation. 

Using two points on a catheter to locate and orient the catheter in space is 
known in the art, as illustrated by Figure 9 in the «853 patent. An embodiment of the 
present invention includes assignable attribute data field for each coordinate, along 
20 with the coordinates of the one or more locations on the catheter. Such a data field 
may store an intensity value corresponding to an amount of ultrasonic energy detected, 
for use in creating false color displays of the catheter image. 

Regardless of the format of the catheter location data, the present invention 
performs a transformation of that location data into the coordinates of the projection(s). 
25 For catheter image files in xheL(x, y, z) format, this process^ may resemble 
: (mathematically) the creation of P(«, v) iromI(x, y, z) , only with a much sparser 
input file.. The projected catheter image(s) P.{u, v) can then be directly superimposed 
on the image data P(«, v). This procedure is repeated, or performed as multiple 

parallel tasks, for each projection orientation. 
20 For catheter Iocation-d2t2-stored as coordinate(s), or coordinate(s) and attribute 

data^ only the individual coordinates need to be transformed into the coordinate 
systems of the projections. That is, there is no need to perform numerical integration 
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on the larger number of data locations making up the catheter image. Such a 
transformation of one or more points to the coordinate system of the projections may 
be performed relatively quickly, in either software or hardware. The output of the 
transformation system may be two (planar) coordinates, for each, point location in a 
5 ■ siven projection, and the Values of the associated attribute data. Mapping a location 
symbol to a CRT display at the location of these two coordinates is straight forward, 
and known to those of ordinary skill in the. art. The attribute data may also be used in 
conjunction with the location symbol. For example, "cross-hairs",- or an arrow symbol, 
could be mapped onto, the CRT display of each projection at the current location of the 

10 - catheter in that proj ection, with the color of the symbol correlated to the magnitude of 
vibration detected in a vibrating catheter. 

An embodiment of the present invention converts catheter image data in the 
form of a J c (.x 9 y, z) into one or more coordinates representing the catheter location 
in order to speed the data processing. The full ultrasound image of the catheter is 

15 sacrificed in such an embodiment, for the ease in processing data, for only a few point 
locations. However, with one or two point locations, and the prior knowledge of the 
catheter geometry, there is no need to directly map the full catheter image to the 
display; all of the information can be carried by, or conveyed by, the symbols mapped 
directly to the monitors. 

20 Unlike the projection data created from the image data/(.r, y 9 r), the 

superimposed catheter image does not require that the image intensities (gray scales) 
be accurately transformed unto the displays, only that the catheter locations be 
accurately transformed. It may be advantageous to modify the intensity of the catheter 
image, relative to the other image data, for display clarity. One embodiment of the 
25 ■ present invention uses the catheter location as the center of projection depth when for 
creating the image for display. 

. Another embodiment of the present invention may use color image data files, 
where I(x> y, z) is replaced by separate color space image files, such as L 
Ic(x, y, z) and I*(x 9 y, z) corresponding to an image in RGB color space. Other 

30 color space formats may also be used, and the present invention is not intended to be 
limited to any particular color space representation. Given this disclosure, those of 
ordinary skill in the art' will recognize how the present invention may be adapted to 
work with such color images; 
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Although various embodiments are specifically illustrated and described herein, 
it will be appreciated that modifications and variations of the present invention are 
covered by the above teachings and are with the purview of the appended claims, 
without departing from spirit and intended scope of the present invention'. Moreover, 
5 the present invention may be implemented in software, firmware as well as in 
hardware,, and in a broad range of computer architectures and is intended to be a 
general approach that includes a broad range of specific implementations. 
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CLAIMS 

What is claimed is: 

1 I. An apparatus for displaying ultrasound volumetric object data and an 

2 interventional device, comprising:' 

3 a volumetric data file for storing a plurality of object intensity values, each of 

4 said intensity values mapped to a set of 3-D coordinates; 

5. a first projection plane for displaying a first projection of said object intensity 

6 values, whereby said first projection plane is defined by a first perpendicular vector; 

7 a first image projector for creating a first projection image on- said first 

8 projection plane from said object intensity values projected along said first 
. 9 perpendicular vector; 

10 an. interventional device locator for locating the interventional device at a 

11 device location mapped to said set of 3-D coordinates; and 

12 a first superimposer for superimposing said device location and said first 

13 projection image to create a first display image. 

1 2. An apparatus in accordance with claim 1, further comprising: 

2 a second projection plane- for displaying a second projection of said object 

3 intensity ' values whereby said second projection plane is defined by "a second 
4' ■ perpendicular vector; 

5 a second image projector for creating a second projection image on said second 

6 projection plane from said object intensity values projected along said second 

7 perpendicular vector; and 

8 a second superimposer for superimposing said device location and said second 

9 . projection image to create a second display image. 

1 3. An apparatus in accordance with claim 2, wherein; 

2 ■ a preferred transformation defines an orientation relationship between said first 

3 perpendicular vector and said second perpendicular vector. 
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1 4. An . apparatus in accordance with claim l, wherein: 

2 said first display im2ge is a view used in common practice by medical 

3 personnel performing a procedure. 

1 5. An apparatus in accordance with claim 2, wherein: 

2 said second display image is a view used in common practice by medical 

3 personnel performing a procedure. 

1 6.. . .An apparatus in accordance with claim 4, wherein: 

2 said first display image is a right anterior oblique (RAO) view of the object. 
1 7. An apparatus in accordance with claim 5,. wherein: 

.2 " said second display image is a left anterior oblique (LAO) view of the object. 

1 8. An appararus in . accordance with claim 4, wherein: 

2 said first, display image is a LAO view of the object. 

1 9. An apparatus in accordance with claim 5, wherein: 
' 2 . said second display image is a RAO view of the object. 

1 10. An apparatus in accordance with claim 1,. further comprising: . 

2 a first image range value for specifying a range of said object intensity values 
.3 in said first display image. 

1 1 1. An apparatus in accordance with claim 2, further comprising: 

2 a second image range value for specifying/a range of said object intensity. 

3 values in said second display image. 

1 12. An apparatus in accordance with claim 10, wherein: 

2 said first image-range is centered on said interventional device location as 

3 measured along said first perpendicular vector. 
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1 13. An apparatus in accordance wiih claim 1 1, wherein: 

2 said second image range is centered on said interventional device location as 

3 measured along said second perpendicular vector. 

1 14. An apparatus in accordance with claim 1, further comprising: 

2 a first display image orientation control for defining said first perpendicular 

3 vector from a user input.. 

1 15. An appararus in accordance with claim 1, wherein: 

2 said superimposer updates, said device location continuously. 

f 16. An apparatus in accordance with claim 15, wherein: 

2 said first image projector updates said object intensity values responsive to a 

. 3 user signal. 

1 17. An apparatus in accordance with, claim 1.5, wherein: 

2 said first image projector updates. said object intensity values in real-time. 

1 18. An apparatus in accordance. with claim 1, wherein: 

2 the interventional device actively transmits a signal and said interventional 

3 device locator image generator is responsive to said signal. 

1 19. An apparatus in accordance with claim 1, further comprising: 

2 an. output file generator for storing said first display image. 

' 1 ■ 20. An apparatus in accordance with claim 2, further comprising: 

2 a third projection plane for displaying a third projection of said object. intensity 

3 values whereby said third. projection plane is defined by a third perpendicular vector; 

4 a third image projector for creating a third projection image on said third 

5 projection plane from said object intensity values', projected along said third 

6 perpendicular vector; and 

7 a third superimposer for superimposing said device location and said third 

8 projection image to create a third display image, 
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1 2i. An apparatus lor cispiaymg uiirasouna voiumeLnc ooject caia ana an 

2 interventional 

3 device, comprising: 

4 a volumetric .data file for storing a plurality of object intensity values, each of 

5 said intensity values .mapped to a set of 4-D coordinates; 

6 a first projection planefor displaying a first projection of said object intensity . 

7 values, whereby said first projection plane is defined by a first perpendicular vector; 

8 a first image projector for creating a first projection image on said first 
5 projection plane from said object intensity values projected along said first 

10 perpendicular vector; 

11 an interventional device locator for locating the interventional device, at a 

12 device location mapped to said set of 4-D coordinates; and 

12 a first superimposer for superimposing said device location and said first 
14 ■ projection image to create a first. display image. 

1 22. An apparatus in accordance with claim 21, wherein: 

2 said first display image is a cine loop. 

1 23. An apparatus in accordance with claim 2 1 , wherein: 

2 said first display image corresponds to a single point in a repetitive sequence. 

1 24. A method of displaying ultrasound volumetric obj ect data and an interventional 

2 device, comprising: 

3 selecting, a plurality of object intensity values mapped to a -set of 3-D 

4 coordinates; • 

5 selecting- a first projection plane for displaying a projection of said object 

6 intensity values,, whereby said first projection plane is defined by a first perpendicular 

7 vector; " 

8 projecting said plurality of object intensity values along said first perpendicular 

9 vector onto said first'pfojection plane; 

10 finding an interventional device location relative to said set of 

11 and ' 

12 ' superimposing said device location and said first proj ection plane onto a first 
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1 25. A method-m accordance with claim 24, fun her comprising: 

2 selecting a second projection plane for displaying a second projection of said 

3 object intensity values projecting a second display, whereby said second projection 

4 plane is defined by a second perpendicular vector; and 

. 5 projecting said plurality pf object, intensity values along said second 

6 perpendicular vector onto said second projection plane; and 

7 superimposing said device location and said second projection plane onto a 

8 second display image. 

1 ' 26. A method in accordance with claim 25, wherein': 

2 selecting a second projection plane is performed using said first perpendicular 

3 vector arid a preferred transformation. 

1. 27. A method in accordance with claim 24, wherein: 

2 selecting said first projection plane conforms to a common practice by medical 

3 personnel performing a procedure. 

1 28. A method in accordance with claim 25, wherein: 

2 selecting said second projection plane conforms to a common practice.- by 

3 medical personnel performing a procedure. 

A method in accordance with claim 27, wherein: . * 
said first projection plane is a right anterior oblique (RAO) view. 

A method in accordance with claim 28," wherein: 
said second projection plane is a left anterior oblique (LAO) view. 

A method in accordance with claim 27, wherein: 
said second projection plane is a LAO view. 
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1 32. A meihod in accordance with claim 25, wherein: 

2 said second projection plane is a RAO view. 

1 33. A method in accordance with. claim 24, further comprising: 

2 selecting a first image. range value for limiting said plurality of object intensity 

3 values in said first display image. 

1 34. A method in accordance with claim 25, further comprising: 

2 selecting a second, image range value for limiting said plurality of object - 

3 intensity values in said second display image. 

1 35. A method in accordance with claim 33, further comprising: 

2 . centering said first display image on said device location; as measured along 

3 said first perpendicular vector. 

1 36.. A method in accordance with claim 34, further comprising: 

2 centering said second display image on said device location, as measured. along 

3 said second perpendicular vector. 

1 37. A method in accordance with claim 24, further comprising: 

2. . repeating said finding of said device location continuously. 

1 38. A. method in accordance with claim 24, further comprising: 

2 selecting a revised plurality of object intensity values in response'to a user 

3 signal. 

1 39. A method in accordance with claim 24, further comprising: 

2 selecting a revised plurality of object, intensity values continuously. 

7 40. A method in accordance with claim 24, further comprising: 

2 selecting a revised piuraliry of object intensity values in response to amotion 

'3 greater than a threshold value. 
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1 41. A method in accordance with ciaim 24, further comprising: 

2 selecting a third projection plane for displaying a third projection of said object 

3 intensity values projecting a third display, whereby said third projection plane is 

4 defined by a third perpendicular vector; and 

5 projecting said plurality of object intensity values along said third perpendicular 

6 ■ vector onto said third projection plane; and 

7 superimposing said device location and said third projection plane onto a third 

8 display image. 

1 42. A method of displayingultrasound volumetric object data and an interventional 

2 device, comprising: 

3 ■■ selecting a plurality of object intensity values mapped to a set of 4-D 

4 coordinates; 

5 selecting a first projection plane for displaying a projection of said object 
5 ' intensity values, whereby said first projection plane is defined by a first perpendicular 

7 vector; 

8 projecting said plurality, of object intensity values along said first perpendicular 

9 vector onto said first projection plane; 

10 finding an interventional device location relative to said set of 4-D coordinates; 

11 ■ and 

12 superimposing said device location and said first projection plane onto a first 

13 display image. 

1 43. A method in accordance with claim 42, wherein: 

2 superimposing said device location is. performed in a cine loop. 

1 44. A method in accordance with : claim 42, wherein: 

2 ■ superimposing said device location.is performed at a single point in a repetitive 

3 sequence. 
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